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c. Interrelationships and comparability of evaluation systems; 

d. Critique of current evaluation systems and gaps, with 
reference to the needs of unit, office /division, directorate and 
Agency levels of management respectively; 

e* Recommendations. 

3. While we should move with all deliberate speed on this action, 

1 think it important that the discussion and resultant recommendations 
be fully informed and responsive to CIA’s unique requirements for flexi- 
bility, security, and adjustment to customer and policy needs. Thus, no 
date is established for this Deputies’ discussion, but you and the Deputies' 
representatives should recommend a date which will give the time needed 
to do a thorough and sensitive study responsive to the Director’s desires 
and appropriate for presentation to him. 


/s/ WEC 
W. E. Colby 

Executive Director- Comptroller 


cc: DO / I 
DD/P 
DD/S&T 
DD/S 

WECsblp 

Distribution: 

Original - IG 
1 - Each Deputy Director 

/T> er 

1 - ExDir 
1 - D/PPB 
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MEMORANDUM FOR: Executive Director-Comptroller 

SUBJECT : CIA Evaluation Systems - Preliminary Observations 


REFERENCES 


(a) Ex Dir Memo to IG dtd 16 Feb 72, Same Subject 

(b) IG Memo to Deputy Directors dtd 3 Mar 72, 

Same Subject 


Having surveyed seventeen of the components that fall within the 
scope of your instructions to the IG-PPB Task Force, we believe \<re can 
now offer some preliminary observations on evaluation procedures pre- 
sently extant in the Agency: 

a. Evaluation systems vary significantly among components and 
directorates. A few offices have devoted considerable resources to 
assessing performance and have developed formal methodologies. M° s t 
offices , however, proceed with differing degrees of informality and 
attention to the problem . 

b. All evaluation efforts depend on individual judgments made 
throughout the command ladder of a given component. The factors 
entering into these judgments are often difficult to specify . They 
include, but are not limited to, such invaluable things as historical 
experience, substantive expertise, sensitivity to current concerns of 
policy-makers, and awareness of actual intelligence requirements. 

c. Formal requirements structures (production schedules, OD's, 

SORS and IGCP lists, etc.) act as a general framework to direct the 
programs of components. Within this framework, managers have consider- 
able freedom in deciding how to allocate their efforts. Some of those 
efforts must be devoted to fulfilling informal, ad hoc requests. With 
the resources that remain available (and the amount of such resources 
varies from component to component) , both collectors and producers self- 
initiate projects which they consider significant. 

These formal and informal directions and self-generated studies 
have been promulgated by diverse sources and have been approved with 
varying degrees of precision and authority. Consequently, they do not 
lend themselves to use as standards or goals against which performance 
may be evaluated. 
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d. For each component, judgments about the value of intelligence 
production are based almost exclusively on an understanding of the 
priorities of their consumers. In turn, this understanding is based 
on common sense, the flow of events, and instructions from the con- 
sumers themselves. Formal priority guidance such as DCID 1/2 is seen 
as either too static or too obvious to be of much use. 

With the exception of the 1974 Program Call, there is no com- 
prehensive mechanism which explicitly or precisely records the value 
judgments of components . Resource allocation decisions cannot be re- 
liably used as indicators of such judgments because they reflect not 
only the relative priority of intelligence objectives, but also the 
feasibility of working against them. 

e. Many components actively solicit consumer feedback in evaluat- 
ing their product. Others, however, feel that such responses are pro 
forma and unreliable, difficult to obtain, and often too late to be of 
practical value. Consequently, they rely on their own judgments. 

When feedback is received, it is generally informal and in- 
frequently systematically recorded. It will usually take the form of 
letters, phone calls, and statements from consumers to office managers. 

The oral briefings and meetings given and attended by intelligence peo- 
ple are a frequent forum for such feedback. 

Often the evaluation of an intelligence product cannot be done 
immediately, but must await the passage of time. In such cases, con- 
sumer feedback is rare, and many worthwhile accomplishments go unnoticed. 
Conversely, products that appear to be valuable in the short run may 
seem far less valuable from a longer-term perspective. 

f. Few components quantify their evaluative judgments . The pri- 
mary exceptions are FE and EUR Divisions in the Clandestine Service. 

Their officers grade raw reports from their field stations and array 
this data in order to assess performance against objectives. Many com- 
ponents compile statistics which describe their output in such terms as 
number of reports disseminated or published; this information, however, 
is not evaluative. 

There is a strong tendency, especially among the production 
officers, to do "binary evaluations." Finished intelligence is judged 
either acceptable or not acceptable for publication. Once accepted, 
no further evaluations are explicitly made. 

In many offices, managers use as the principal measure of satis- 
factory performance the absence of complaints from their superiors at the 
directorate level or from close consumers. It is unusual to find an office 
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that has promulgated distinct guidelines for striving toward excellence 
in output. Rather, most effort seems to be devoted to- managing input. 

In turn, because the accounting system stresses living within the budget, 
some managers view the husbanding of their resources as a primary basis 
on which they will be evaluated. 

g. No present evaluation effort permits a systematic comparison 
of the performance of a single component over several years or the per- 
formance of several different components in a single year . Moreover , 
within offices, managers do not make explicit their judgments about the 
relative performance of subdivisions. 

h. Both producers and collectors have expressed a distrust of any 
formal evaluation system. They are concerned that such a system would 
not reflect their unique problems and could be misused. They feel that 
their evaluative efforts are either generally satisfactory for their 
needs or that major improvements are not possible. 
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Inspector General 


Approved For Release 2001/11/22 : CJA-RPP80B01086A000900090026-7 



Approved For Relj 


ie 2001/11/22.' ; CiA-RDP80B01086A0i 


100090026-7 


Rationale for Development of Evaluation Procedures 


An anomalous feature of the CIA is the fact that we have developed 
a successful way of doing things and at the same time have developed no 
effective way of justifying the things we do. It has long been assumed 
that evaluations are best done informally, that they are the business 
only of the individual component or directorate, and that any form of 
centralized evaluation is undesirable, if not impossible. A new set of 
pressures, however, is calling these assumptions into question. 

The initial pressure for change is external. The people who allo- 
cate. resources — both in 0MB and the Congress — will not be satisfied 
much longer with the kind of program defenses we have offered in the^ 
past. The new people in these organizations (young Senators and their 
staffs, budget examiners, systems analysts, etc.) continue to chip away 
at our traditional special status. They are deeply concerned about pro- 
ductivity throughout the government. Consequently, they want data — 
not just pictures and stories — which will enable them to know how our 
collection, production, and support components are performing in rela- 
tion to each other and to various intelligence targets. It is not 
possible to construct this overview of the Agency from the separate, 
necessarily limited evaluation systems that now exist in each component. 

Other government agencies either provide this kind of central data 
or are preparing to do so. The Department of State, for example, has 
instituted an evaluation group under its IG staff. Within the IC struc- 
ture, the NSCIC, the Net Assessment Group, and the Product Improvement 
Group of the IC Staff will be considering this problem. The Agency is 
in no position to respond to requests for such data, and our position 
could become increasingly untenable if nothing is done. It is probably 
true that we have performed better than many of the organizations that 
have extensive evaluation systems. This past performance, however, 
will not be accepted as a reason for now failing to develop an appro- 
priate evaluation system in line with the rest of government. 

If we can make it effective, this kind of system can help us with 
our internal management problems. It is clear that all of us rely on ^ 
the overall evaluations of our managers. Although this practice is both 
inevitable and desirable, we should have some wa y of checking on the 
basis for those evaluations. It is true that we would knox-7 of excep- 
tionally poor performance, both by the complaints of customers and by 
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our own warning signals. The nature of bureaucracy, however, can con- 
ceal mediocre performance or unnecessary effort for unacceptably long 
periods. 

There are, however, more significant reasons for moving ahead in 
this area. Each year resource decisions reflect our judgment on the 
best mix between programs. Given the present size of the Agency, how- 
ever, we cannot make those decisions on any methodical basis other than 
broad parameters of obvious success or failure, or minor percentage 
changes from current levels allocated by bureaucratic proportion. If 
we had more precise feedback — and feedback that permitted intra— and 
inter-program comparisons — we would know better where additional 
money should be spent and where it could possibly be saved. With, this 
kind of information as a "check," we could continue and even expand our 
policy of decentralized management. 

The usual objection to development of an evaluation system is that 
many of our activities and values are intangible, variable, and unquan— 
tifiable. This is not a barrier, it is a challenge , and one which can 
be and has been met in a number of cases. In part, a system can assign 
values to intangibles. In part, the system can represent its own limi- 
tations and serve as a management tool with, no pretentions to absolute 
truth. In part, it must be recognized that evaluations are made at 
present on each occasion resources are allocated, and that a system is 
an attempt at improvement rather than an assertion of perfection. 

Evaluation systems of course have the potential danger of becoming 
the master rather than the servant of managers. They can become inflex^ 
ible and stifle initiative at lower levels. This should not worry us. 
The Directorates and the Offices themselves should play an important 
role in making the evaluations. Moreover, none of us is going to 
placidly accept the results of evaluations which, run counter to common 
sense. Instead, we will ask questions, and in so doing make explicit 
many of our own assumptions. As a result, the evaluation procedures 
will have to be continually modified 5 but in some cases, so will our 
common sense judgments. This impetus for questioning and rethinking 
decisions would probably be the primary advantage of more systematic 
evaluations. In other words, an evaluation system could never provide 
answers to the complex problems we face, but it could raise questions 
we might not recognize without it. 
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Proposal for Future Action on Evaluation 


A compatible set of evaluation systems in CIA must reflect the 
unique aspects of the Agency as a whole and of its individual com- 
ponents. At the same time, the systems must provide some common de- 
nominator for analyzing output which will satisfy both our internal 
and external management needs. 

To develop the required procedures, a task force should be estab- 
lished, first to determine the precise requirements and then to insure 
that the systems established meet those requirements. This task force 
should have as its chairman a representative from the Office of the 
Inspector General and as vice-chairman, a representative from the Office 
of Planning, Programming, and Budgeting. Each directorate would have 
one member. These men would be able to explain the unique aspects of 
and represent the interests of their organizations. They should remem- 
ber, however, that their primary responsibility is to provide the DCI 
with a means for evaluating the performance not only of their own di- 
rectorate, but of the entire Agency. This task force will report its 
findings to the Inspector General to make recommendations for implemen- 
tation to the Executive Director-Comptroller. Clearly, developing this 
kind of evaluation structure will be a formidable task, and there will 
have to be continual revision over the years. Nevertheless, I would 
hope to have systems functioning by the beginning of CY 1973. 
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